Searching P,AJ 



Page 1 of 2 



PATENT ABSTRACTS OF JAPAN 



(1 1 )Publication number : 1 0-040395 

(43)Date of publication of application : 13.02.1998 



(51)lnt.CI. 




G06T 9/20 




(21)Application number 


08-190471 


(71)Applicant 


HITACHI LTD 


(22)Date of filing : 


19.07.1996 


(72) Inventor : 


KATO MAKOTO 



i«-Jt*w 



















mm? is- AHfib«jtiflys 



^103 



(54) METHOD FOR EXTRACTING OUTLINE OF DYNAMIC IMAGE 
(57)Abstract: 

PROBLEM TO BE SOLVED: To surely and efficiently 
extract the outline of the image of a rapidly moving 
object from a dynamic image. 
SOLUTION: When a user selects two discontinuous 
reference frames from a picture in plural frames 
displayed as a list on a display (103), each outline of the 
image of an object is extracted from the picture in each 
reference frame (104). When the user indicates the 
proper number of characteristic points on the outline of 
the image of the object in the image in each reference 
frame, and makes them correspond to each other (105), 
the outline of the image of the object in an intermediate 
frame between the two reference frames is presumed 
from the coordinates of the characteristic points in the 
image in each reference frame (106). Then, this is used as an outline for starting retrieval. 
Then, the retrieval of the image in each intermediate frame is started from the retrieval 
starting outline, and the actual outline of the image of the object is extracted. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 

DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] this invention relates to the picture extraction method 
that the border line of the picture of the intense move body of movement can be certainly extracted out 
of a dynamic image. 
[0002] 

[Description of the Prior Art] Let it be fundamental technology to extract the field of an analysis object 
from a dynamic image in the dynamic image analysis used widely in various fields, such as medicine 
and meteorology. Generally such technology is called picture segmentation technology, and the 
boundary detection which extracts the border line of the picture of an analysis object is known by using 
the changing point of the features (for example, luminosity etc.) of a picture as an edge and connecting 
[ detect, emphasize and ] as the way method, this Boundary Detection -- being Related -- Computer 
Vision, Prentice-Hall, and Inc. -- it is Indicated by 1982 "Chapter 4 Boundary Detection" (D. H.Ballard, 
C.B.Brown Work) The edge detection which mentions as an example the case where the monochrome ' 
static image f (x y) is made into an object picture, and is hereafter indicated by this about the edge 
detection characteristic of this kind of technique is explained. 

[0003] It asks for the large place of the color between the pixels adjoined in a picture, or the difference 
of picture concentration as an edge fundamentally, and asks for a profile by connecting this. Here, the 
word "edge" is a segment in a certain meaning which exists in a picture, and a profile becomes a closed 
contour, when the body showing the appearance of the body which exists in a picture does not hide into 
a picture but is showing. Many edges exist also on a profile besides it. Moreover, in the technique of a 
usual image processing conversely like the edge-detection operator who states later, not all on a profile 
are recognized as an edge. 

[0004] Being used in order to ask for an edge is as follows. 

[0005] In the case of monochrome shade picture, i and j pixels are expressed with a (i, j). The direction 
of i corresponds to a longitudinal direction, and the direction of j corresponds to lengthwise. 
[0006] Primary differential longitudinal direction : (1) a(i+l, j)-a (i, j), lengthwise:a(i, j+l)-a (i, j), 
square [ of absolute value: (a(i+l j)-a (i, j)) ] ((2) quadratic-differential longitudinal direction:, such as a 
square root of the square [ of a(i, j+l)-a (i, j) ] sum, -- a(i+l, j)-2a(i, j)+a (i-1, j) --) Lengthwise: The sum 
of the last both directions may be especially called Laplacian here [, such as sum:a(i+l, j)+a(i-l j)+a(i, 
j+l)+a(i, j-l)-4of a(i, j+l)-2a(i, j)+a (i, j-1) and both directions a (i, j), ]. This Laplacian **2 It is 
computed in fact by the convolution operation using the operator of the three-line three trains shown 
below. 

[0007] 
0 1 0 

1-4 1 

0 1 0 
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That is, Laplacian **2 will be expressed like the following formula as consequent. 
[0008] 

**2 =a(x y-l)+a(x-l, y)-4a (x y) 
+ a(x+l, y) +a (x y+1) - (formula 1) 

In addition, as an operator who uses for the convolution operation performed in the case of an edge 
detection, the thing of three-line three trains may necessarily be used, and the thing of a n line m train (n 
and m are the natural number) may be used. 

[0009] By the way, the edge detected by such technique has many way piece ****** cases by the 
influence of noise etc. Then, above Computer Vision Prentice-Hall Inc. - in order to Connect ******** 
Edge Smoothly, Using the Edge Pursuing Method is Indicated by 1982 "Chapter 4 Boundary 
Detection" (D. H.Ballard, C.B.Brown Work) (P. 131 to P. 137 Reference) 

[0010] As edge detections other than this, the picture segmentation method given in JP,7-121710 A is 
learned, for example. This Picture Segmentation Method Has the Feature in Receiving Specification of 
Search Start Pixel on Profile of Picture of Analysis Object, and Specification of Direction that Edge 
Search is Advanced, from User Side unlike Edge Detection above Computer Vision, Prentice-Hall and 
Given in Inc. 1982 "Chapter 4 Boundary Detection" (D. H.Ballard, C.B.Brown Work). In addition, 'it is 
also characteristic that a user can extract as an edge the border line which drew with the mouse etc. 
according to directions of a user, that is, automatic extracting only of the edge of the analysis object 
from the picture in which the edge became intricate certainly and intricately efficiently [ be / it / more 
markedly / than the case where edge search is arbitrarily advanced since a user can determine the edge 
of an analysis object intentionally when that a user gives the indicator of edge search intentionally fails 
in connection of an edge on ****** / alike and ] can be carried out 

[001 1] By the way, when detecting the edge of an analysis object from a dynamic image, technique is 
not applied like the edge detection in the above-mentioned static image about all the frames that 
constitute a dynamic image. In order to process a vast quantity of data efficiently, usually 
"SNAKES:Active Contour Models" (M. Kess, A.Witkin, D.Terzopoulos) International Journal of 
Computer Vision and vol.1 No. - 4 and 1988 As indicated Although the edge of an analysis object is 
detected according to the same technique as the edge detection in the above-mentioned static image only 
from the frame of the beginning of the frames which constitute a dynamic image The technique of 
detecting the edge of an analysis object is adopted by performing edge search about the frame which 
continues after that by making into a search start point the edge of the analysis object extracted from the 
last frame. 
[0012] 

[Problem(s) to be Solved by the Invention] However, the technique adopted when detecting the edge of 
an analysis object from the above-mentioned dynamic image had the fault that detection of the edge of 
the intense analysis object of movement could not be coped with. That is, if the movement of an analysis 
object is intense, since the position of an analysis object is remarkably different mutually in many cases 
in a continuous frame, the edge of an analysis object cannot necessarily be detected for the edge of the 
analysis object extracted from the last frame as a search start point. 

[0013] Then, this invention aims at offering the dynamic-image extraction method that the picture of the 
intense move body of movement can be extracted certainly and efficiently out of a dynamic image. 
[0014] 

[Means for Solving the Problem] In order to solve the above-mentioned technical problem, this 
invention is the picture extraction method of extracting the border line of an objective picture from the 
picture of two or more frames one by one. The first step which receives specification of two or more 
focus on the border line of the picture of the aforementioned body corresponding to mutual between the 
pictures of two criteria frames which are not followed of two or more aforementioned frames, It is based 
on the position of two or more focus on the border line of the picture of the aforementioned body 
corresponding to mutual between the pictures of each aforementioned criteria frame. The second step 
which computes the search start line which starts search of the border line of the picture of the 
aforementioned body in the picture of the intermediate frame pinched by two aforementioned criteria 
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frames, respective y, In the picture of the intermediate frame pinched by two aforementioned criteria 
frames, search of the border line of the picture of the aforementioned body is started from the 
aforementioned search start line, and the picture extraction method characterized by including the third 
[001 5] CXtraCtS b ° rder ° f Pi ° tUre ° f aforementioned bod y one b y one is offered. 

[Embodiments of the Invention] Hereafter, one gestalt of operation concerning this invention is 
explained, referring to an attached drawing. 

[0016] First, drawing! explains the basic composition of the hardware of the image processing system 
concerning the gestalt of this operation. 

[0017] The input unit with which this image processing system receives the input from a user The 
picture input device 308 which inputs digital dynamic-image data for every frame, The display unit 305 
which displays a dynamic image (for example, a CRT monitor, a liquid crystal display, etc.), The 
auxiliary memory 307 which stores the various programs which defined data processing performed by 
the after-mentioned digital dynamic-image data [ which were inputted from the picture input device 
308 ], and segmentation processing, The processor 301 which performs the below-mentioned 
segmentation processing, and the buffer memory 302 for processor 301, The frame memory 303 for 
processor 301, and the input/output control processor 306 which connects auxiliary memory 307, a 
picture input device 308, and an input unit 3 10,309 to a processor 301, It has the display controller 304 
who changes into an RGB code the image data stored in the frame buffer 303, and outputs to a display 
unit 305, and the system bus 3 1 1 which connects these each part mutually. 

[0018] The display unit 305 which displays a dynamic image has the property (namely, each red bluish 
green a 1000x1 000-pixel display, 256 gradation, 16,700,000 color displays) made general as this kind of 
a display unit. When using other display units which have a different property from this instead of this, 
it is desirable to change picture I/O device 308 used now into other picture input devices which have the 
performance in which it can respond to the property of a new display unit if needed. 
[0019] The input unit which receives the input from a user consists of mouse 310 grades which receive 
specification of the arbitrary positions on the keyboard 309 which receives selection of the function 
assigned to each key, and the screen screen of a display unit 305. 

[0020] Therefore, after the user moved the cursor currently displayed on the screen screen of a display 
unit 305 using the mouse 310, Furthermore, the button carried in the mouse 310 (with the gestalt of this 
operation) Out of two or more frames (refer to drawin g 5 ) which constitute the dynamic image by 
which it was indicated by the list from a state where it stood in a line on the screen screen of a display 
unit 305 by clicking the predetermined button of the two buttons at time series The frame which 
constitutes a desired scene can be specified (after-mentioned). Hereafter, the mouse operation by the 
user at this time is called "pick by the mouse 310." Moreover, the state where the button carried in the 
mouse 310 is pushed will be called "mouse-on state", and this will call "mouse-off state" the state where 
the button conversely carried in the mouse 3 1 0 is not pushed. 

[0021] In addition, although the mouse 3 10 is used with the gestalt of this operation as an input unit for 
specifying the position on the screen screen of a display unit, it is not necessary to necessarily use a 
mouse and other locator equipments (for example, joy stick etc.) may be used as substitution of a mouse 
310. Moreover, it is desirable for it not to be necessary about the operating instruction of a mouse 310 to 
adopt an operating instruction as explained here, and to adopt the operating instruction which added a 
suitable change in consideration of a user's operability. 

[0022] Moreover, a user can input the numeric value assigned to each key, the command which directs 
program execution by choosing a suitable key out of the key of a keyboard 309. 
[0023] The field 401 which memorizes predetermined data with starting of a system as shown in the 
buffer memory 302 for processor 301 at drawin g 2 , i.e., the program field which memorizes the 
predetermined program stored in auxiliary memory 307, the operating system field (un-illustrating) 
which memorizes the various programs which constitute an operating system, and the data area 402 
which memorizes the various data needed in the below-mentioned segmentation processing are secured. 
[0024] And each field included to the program field 401 The field 403 which memorizes the whole 
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?w Sr,. Wh L 1Ch defmed the fl0W 0f the below-mentioned segmentation processing, respectively 
The field 404 which memorizes the criteria frame profile extraction program which defined the criteria 
frame profile extraction processing performed by below-mentioned segmentation being under 
processing, The field 406 which memorizes the profile search program which defined the profile search 
processing performed by below-mentioned segmentation being under processing, It is assigned as a field 
405 which memorizes the search start profile calculation program which defined the search start profile 
calculation processing performed by below-mentioned segmentation being under processing 
[0025] On the other hand, each field included in the data area 402 is digital dynamic-image data () 
stored in auxiliary memory 307, respectively. Or the dynamic-image data area 407 which memorizes the 
digital dynamic-image data inputted from the picture input device 308 and the border-line data area 408 
which memorizes the border-line data extracted by profile search processing performed by below- 
mentioned segmentation being under processing, It is assigned as the kernel pattern field 409 which 
memorizes the differential type operator used for the convolution operation of an edge detection, and a 
working area 410 which memorizes temporary data created in the below-mentioned segmentation 
processing. 

[0026] Above, the explanation about the composition of the hardware of the image processing system 
concerning the gestalt of this operation is finished. In addition, as the processing section equivalent to a 
processor 301, buffer memory 302, and an input/output processor 306, although the information 
processor (a workstation, personal computer) of a stand-alone type is used, it is not necessary to 
necessarily do in this way with the gestalt of this operation. For example, a network system constitutes 
this image processing system, and it may be made to perform data processing in the below-mentioned 
segmentation processing etc. in parallel with many computer resources. Moreover, it is not necessary to 
necessarily use the information processor used with the gestalt of this operation and the information 
processor of architecture of the same kind, i.e., the information processor of single bus architecture with 
which a processor 301, buffer memory 302, and an input/output processor 306 share one system bus. For 
example, you may use the information processor which connected between a processor 301 and 
input/output processors 306 by the individual system bus between a processor 301 and buffer memory 
302, respectively. 

[0027] The segmentation processing which a processor 301 performs by drawing 3 and drawing 4 
hereafter according to the whole control program memorized to the field 403 of buffer memory 302 is 
explained. 

[0028] If a user inputs a predetermined command using an input unit, the segmentation processing 
shown in drawjngJ3 will be started. That is, in Step 101, if the digital dynamic-image data (or digital 
dynamic-image data inputted from the picture input device 308) stored in auxiliary memory 307 are 
memorized by the dynamic-image data area 407 of buffer memory 302, a display controller 304 stores 
this in a frame buffer 303, after performing a predetermined image processing to the digital dynamic- 
image data memorized by the dynamic-image data area 407 of buffer memory 302 at this time. And after 
changing into an RGB code the digital dynamic-image data stored in the frame buffer 303, it outputs to a 
display unit 305. 

[0029] Consequently, in Step 102, on the screen screen of a display unit 305, as shown in drawing 5 , a 
list indication of the picture of the frame which constitutes a dynamic image is given in the state where it 
ranked with time series. In addition, as long as it is the case where it is not necessary to necessarily 
display the picture of all the frames that constitute a dynamic image for example, and the body with 
comparatively blunt movement is being used as the main photographic subjects, you may be made to 
indicate the picture in every several frames by list. 

[0030] Then, in Step 103, if a user specifies the picture of the head frame Fl of a desired scene, and the 
tail frame Fn, respectively out of the picture of the frame by which it was indicated by the list on the 
screen screen of a display unit 305 with "the pick by the mouse 310", the head frame Fl and the tail 
frame Fn which were specified at this time will be set up as a criteria frame, then, in Step 104, a 
processor 301 performs the edge detection explained in the column of the conventional technology using 
the differential type operator memorized at the criteria frame profile extraction processing 409, i.e., the 
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kernel pattern field of buffer memory 302, according to the criteria frame profile extraction processing 
program memorized to the field 404 of buffer memory 302, and extracts the border line which is the 
picture of Body B from the picture of two criteria frames Fl and Fn, respectively in addition, the need 
that criteria frame profile extraction processing is not necessarily based on the edge detection explained 
in the column of the conventional technology may be field division which there is not, for example is 
known as other way methods of segmentation technology And the border-line data showing the border 
line of the picture of the body B extracted from the picture of two criteria frames Fl and Fn are 
memorized to the border-line data area 408 of buffer memory 302, respectively. Then after a display 
controller 304 compounds the border-line data showing the border line of the picture of the body B 
extracted from the picture of two criteria frames Fl and Fn, and the image data showing the picture of 
two criteria frames Fl and Fn, respectively, he performs the same processing as Step 102, and outputs to 
a display unit 305. Consequently, on the screen screen of a display unit 305, the picture of the criteria 
frames Fl and Fn which gave a synthetic indication of the border lines Bl and Bn of the picture of the 
newly extracted body B is displayed. 

[003 1] As shown in drawing_6 , a user in Step 105 then, with "the pick by the mouse 310" In the picture 
of two criteria frames Fl and Fn, only the suitable number (n-1) points to the border line Bl of the 
picture of Body B, the focus on Bn (Al, An), and (Nl, Nn), respectively. If each focus (Al, Nl), 
(An, Nn) are matched mutually, it will set to Step 106. a processor 301 Recursive processing shown 
in drawing_4 is performed, and the border line of the picture of Body B is extracted from the 
intermediate frame F3 between two criteria frames Fl and Fn, the picture of Fn-1 one by one. 
Namely, a processor 301 performs the following processings one by one by making each intermediate 
frame F2, the picture of Fn-1 into a processing-object picture. 

[0032] In Step 201 first, a processor 301 Search start profile calculation is performed according to the 
search start profile calculation program indicated to the field 405 of buffer memory. The coordinate of 
each focus Kl, Kn on the border line of the body B contained in the picture of the k-th intermediate 
frame Fk which is a new processing object is presumed, furthermore, between each presumed focus Kl, 
Kn -- digital ones - a segment -- it describes and considers as the search start border line 702 which' 
should make this the origin which searches for the border line of Body B in the picture of an 
intermediate frame Fk (refer to drawin g 7 ) Specifically, as shown below, according to a formula 2, the 
coordinate (Xt, Yt) (however, Kt<n) showing the position of each focus Kl, Kn on the profile of the 
body B contained in the picture of the k-th intermediate frame Fk is presumed first 
[0033] 

(Xt, Yt)={(n-k)(X't, Y't)+(k-l) 
(X"t, Y"t)}/(n-l)- (formula 2) 

It is the coordinate which expresses the position of each focus (Al, Nl) corresponding to mutual, 
(An, Nn) between here and (X't, Y't) (X"t, Y"t) (however, Kt<n) the profile of the body B contained in 
the picture of two criteria frames Fl and Fn. 

[0034] In the picture of two criteria frames Fl and Fn, respectively furthermore, by the edge detection 
Two points are set in the picture of the k-th intermediate frame Fk using the coordinate of AAs and NNs, 
and a formula 2. the adjoining focus (As, As-1), the border line Bl between (Ns, Ns-1), and the points 
AAs and NNs on Bn - extracting - this ~ The coordinate of the focus Ks adjoined on the border line of 
the picture of Body b and the point (it is hereafter called a midpoint) between Ks-1 is presumed. And 
such processing is repeated, and in order to create a reliable search start border line, the coordinate of the 
midpoint of sufficient number is presumed. In addition, with the gestalt of this operation, seven 
midpoints were presumed for every adjoining focus Ks and Ks-between in consideration of the 
adjoining focus Ks, the interval of Ks-1, etc. 

[0035] then, digital one of common knowledge, such as a DDA (digital differential analyzer) algorithm, 
- a segment ~ according to a depiction algorithm, between Focus Kl, Kn and the midpoints which 
were presumed is drawn, respectively, and let this be the search start profile 702 in the picture of the k- 
th intermediate frame Fk (refer to drawin g 7 ) In addition, the DDA algorithm mentioned as an example 
here is simplest algorithm as an algorithm which draws between dispersed points in digital one, and is 
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indicated by 3 [ besides Computer Graphics-principles and practice- and Addison- Wesley 1990 
J.D.Foley ] excellent book (p73 -p74) about the detail. 

[0036] Then, a processor 301 performs profile search processing in Step 202 according to the profile 
search program memorized to the field 406 of buffer memory. That is, in the picture of the k-th 
intermediate frame Fk, the point which exists really the search start point set up for every interval 
suitable on the search start profile 702 computed at Step 201 on the border line Bk of the picture of the 
actual body B from the picture of the k-th intermediate frame Fk by performing an edge detection as an 
origin of 1 is extracted, in addition, "SNAKES: Active Contour Models" (M. Kass A Witkin D 
Terzopoulos) Intwernational Journal of ComputerVision explained in the column of the conventional 
technology as an edge detection performed at this time, for example and vol.1 No. - what is necessary is 
just to adopt the well-known technique, such as technique, like the edge search method indicated by 4 
and 1988 And same processing is performed about all search start points, respectively and the point 
which exists really on the border line of the picture of the border line of Body B is extracted one by one 
And between the extracted points is drawn according to well-known drawing algorithms, such as a DDA 
(digital differential analyzer) algorithm, and let this be the border line Bk (to refer to drawin g 7 ) of the 
picture of the border line of the body B contained in the picture of the k-th intermediate frame Fk. And 
the border-line data showing this border line Bk are memorized to the border-line data area 408 of buffer 
memory 302. 

[0037] A processor 301 performs recursive processing shown in drawin g 2 . now, from all the 
intermediate frames F2, Fn-1 If it finishes extracting the border line B3 of the picture of the border 
line of Body B, Bn-1, it will set to Step 107, respectively, a display controller 304 The border-line 
data showing the border line of the picture of the body B extracted from the picture of each frames Fl, 
Fn memorized by the border-line data area 408 of buffer memory 302, every - after compounding ' 
Fl, the image data showing the picture of Fn, respectively, the same processing as Step 102 is 
performed, and it outputs to a display unit 305 Consequently, on the screen screen of a display unit 305, 
a list indication of the picture of each frames F 1 and Fn by which the border lines B 1 , . . . , Bn of the 
picture of Body B were compounded is given in the state where it ranked with time series. 
[0038] Thus, since the border line of the picture of the body contained in the picture of an intermediate 
frame is presumed from the border line of the picture of the body contained in the picture of two criteria 
frames and this is made into the origin of the search in the picture of each intermediate frame, as shown 
in djawing.7. , in the picture of each intermediate frame, search can be started from the position near the 
**** of the border line of the picture of an actual body, respectively. That is, even if it is the intense 
body of movement, the border line of the picture can be extracted efficiently and certainly. 
[0039] Above, the explanation about the segmentation processing concerning the gestalt of this 
operation is finished. 

[0040] In addition, with the gestalt of this operation, although two frames are set up as a criteria frame, it 
is not necessary to necessarily do in this way. For example, when objective movement changes, a 
criteria frame is further set up into an intermediate frame, and it may be made to perform the above- 
mentioned segmentation processing for every scene which has regularity in objective movement. 
[0041] Moreover, although drawing by the DDA algorithm is used with the gestalt of this operation in 
case the objective border line and objective search start profile of a picture are computed, you may use 
the digital alignment drawing algorithm which does not need to use this, for example, not necessarily 
draws a spline curve in digital one. When setting up three or more criteria frames, a suitable multi-term 
function with a degree lower the first order than the number of sheets of a criteria frame is created, and 
you may make it use this as a interpolation function. 

[0042] In addition, a dynamic image may be what photos the same body simultaneously from a different 
direction from a different position using two or more cameras, and compounded and created this 
[0043] 

[Effect of the Invention] According to the dynamic-image extraction method concerning this invention, 
the border line of the picture of the intense body of movement can be extracted from a dynamic image ' 
efficiently and certainly. 
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CLAIMS 



[Claim(s)] 

[Claim 1] The picture extraction method characterized by providing the following of extracting the 
border line of an objective picture from the picture of two or more frames one by one. The first step 
which receives specification of two or more focus on the border line of the picture of the aforementioned 
body corresponding to mutual between the pictures of two criteria frames which are not followed of the 
frames of front plurality. The second step which computes the search start line which starts search of the 
border line of the picture of the aforementioned body in the picture of the intermediate frame pinched by 
two aforementioned criteria frames based on the position of two or more focus on the border line of the 
picture of the aforementioned body corresponding to mutual between the pictures of each 
aforementioned criteria frame, respectively. The third step which starts search of the border line of the 
picture of the aforementioned body from the aforementioned search start line in the picture of the 
intermediate frame pinched by two aforementioned criteria frames, and extracts the border line of the 
picture of the aforementioned body one by one. 

[Claim 2] The picture extraction method characterized by providing the following of extracting the 
border line of an objective picture from the picture of two or more frames one by one. The first step 
which receives specification of two or more focus on the border line of the picture of the aforementioned 
body corresponding to mutual between the pictures of two or more criteria frames which are not 
followed of the frames of front plurality. The second step which computes the search start line which 
starts search of the border line of the picture of the aforementioned body in the picture of the 
intermediate frame pinched by the criteria frame of order based on the position of two or more focus on 
the border line of the picture of the aforementioned body corresponding to mutual between the pictures 
of the criteria frame of order, respectively. The third step which starts search of the border line of the 
picture of the aforementioned body from the aforementioned search start line in the picture of the 
intermediate frame pinched by the criteria frame of order, and extracts the border line of the picture of 
the aforementioned body one by one. 

[Claim 3] The picture extraction method according to claim 2 characterized by providing the following. 
The second step of the above is two or more focus on the border line of the picture of the 
aforementioned body contained in the picture of the criteria frame of 1 . the N which is the number of the 
intermediate frames into which between two or more focus on the border line of the picture of the 
aforementioned body contained in the next criteria frame of the criteria frame of the above 1 
corresponding to each focus of two or more aforementioned focus was inserted between the criteria 
frame of the above 1, and the next criteria frame of the criteria frame of the above 1, respectively - 1 -- 
the step which computes the internally dividing point which divides into the individual which are many 
numbers (N+l) Two or more focus on the border line of the picture of the aforementioned body 
contained in the picture of the criteria frame of the above 1 of the aforementioned internally dividing 
points. About between two or more focus on the border line of the picture of the aforementioned body 
contained in the next criteria frame of the criteria frame of the above 1 corresponding to each focus of 
two or more aforementioned focus, it is J. : (N-J +1) (J) The step which makes the line which 
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interpolated and obtained between the internally dividing points currently divided into the natural 
number below N using the predetermined interpolation function the search start line of the intermediate 
frame Jth after the criteria frame of the above 1 . 

[Claim 4] | It is the picture extraction method that it is the picture extraction method according to claim 3 
and the aforementioned predetermined interpolation function is characterized by being the multi-term ' 
function of a low degree [ primary ] rather than the number of the aforementioned criteria frames. 
[Claim 5] It is the picture extraction method which is the picture extraction method according to claim 3 
and is characterized by the aforementioned predetermined interpolation function being a spline function.' 

[Claim 6] It is the picture extraction method characterized by being the picture extraction method 
according to claim 1, 2, 3, 4, or 5, and two or more aforementioned frames being constituted by the 
frame which photoed the aforementioned body from arbitrary positions and arbitrary directions using 
two or more cameras. 



[Translation done.] 



http://www4.ipdl,jpo.go,jp/cgi-bin/tran . web_cgi_ejje?u=http%3A%2F 0 /o2Fwww6.ipdl,jpo.go.... 2/6/03 



Drawing selection | drawing 1 



Page 1 of 1 




30a 



[Translation done.] 



http://www4.ipdl.jpo.go.jp/cgi-bin/tran_web_cgi_ejje 



2/6/03 



Page 1 of 



Drawing selection {drawing 2 



a 



m2 











403 












404 












405 






406 

— ' 
























407 






« — J 






408 ' 












409 












410 













,4017Q77A*^M 



402 



[Translation done.] 



http://www4.ipdl.jpo.go.jp/cgi-bin/tran web_cgi_ejje 



2/6/03 



Page 1 of 1 



Drawing selection | drawing 3 



1 



,102 



T 



,105 



,106 



,107 



[Translation done.] 



http://www4.ipdl ,jpo.go,jp/cgi-bin/tran_web_cgi_ejje 



2/6/03 



Page 1 of 1 



Drawing selection (drawing 4 



n 



04 




202 



[Translation done.] 



http://www4. ipdl .jpo.go. jp/cgi-bin/tran_web_cgi_ej j e 



2/6/03 



Page 1 of 1 



Drawing selection jdrawing 5* 



H 




[Translation done.] 



http://ww4jpdljpo.go.jp/cgi-bin/tran_web_cgi_ejje 2/6/03 



Page 1 of 1 



Drawing selection | drawing 6~ 



D 




u- a Fn 



[Translation done.] 



http ://www4 . ipdl ,j po . go ,j p/cgi-bin/tran_web_cgi_e j j e 



2/6/0] 



Drawing selection [ drawing 7 



i 




Page 1 of 1 



[Translation done.] 



http://www4.ipdl,jpo.go,ip/cgi-bin/tran_web_cgi_ejje 



2/6/03 



JP10-40395-A 



THOIVISOM 

- * 

DERWENT 



MACHINE-ASSISTED TRANSLATION (MAT): 



(19) mnm] 

a*m¥fW!r (j p) 

(12) [^SHIgiJ] 

'Amm^m (a) 
(11) i&rm^) 

#il¥ 10-40395 
(43) IAMB] 

¥^10^ (1 9 9 8) 2R1 
3 B 

(54) imm<D%m 

(51) im^m^mmem 

G06T 9/20 

[F I] 
G06F 15/70 335 

itmrn^m 6 
[mmm\ ol 

(21) wm&%] 

I¥ 8-190471 



(22) [tune] 

« 8f (1 9 9 6) 7^19 
B 

(71) [ttJIBA] 



(19)[ISSUINGCOUNTRY] 

Japan Patent Office (JP) 



Laid-open (Kokai) patent application number 
(A) 

(11)[UNEXAMINEDPATENTN UMBER] 

Unexamined-Japanese-Patent No. 10-40395 

(43)[DATEOFFIRSTPUBLICATION] 

Heisei 10 (1998) February 13 



(54)[TITLE] 

Extracting method of the outline of a dynamic 
image 

(51)[IPC] 

G06T 9/20 

[Fl] 

G06F15/70 335 

[EXAMINATIONREQUEST] UNREQUESTED 
[NUMBEROFCLAIMS] 6 
[Application form] OL 
[NUMBEROFPAGES] 8 

(21 )[APPLICATION NUMBER] 

Japanese Patent Application No. 8-190471 

(22)[DATEOFFILING] 

Heisei 8 (1996) July 19 

(71 )[PATENTEE/ ASSIGNEE] 



03/02/12 



1/33 



(C) DERWENT 



JP10-40395-A 



THOMSON 

* - 

DERWEIMT 



r^l [IDCODE] 

000005108 000005108 

*S^tt P Hitachi, Ltd. 

l&ffiXftgffi] [ADDRESS] 

=Ht ffl e# ffl mmi? m t 

(72) [|g^#] (72)[INVENTOR] 



^JP0 $£ Kato Makoto 

[ftfjfXfi^Fjf] [ADDRESS] 
0 9 9fl «^ttB^|!if^ 

(74) [ftSA] (74)[PATENTAGENT] 

[#S±] [PATENTATTORNEY] 

[ft£Xfi£#] sffl ^Pi 1 Tomita Kazuko 

(57) imtb] (57)[SUMMARY] 



!] [SUBJECT] 

fflsWH&fab. W)H< <DM^^$)fc Tne contourjine of the image of a rapidly 

»M^W$§$l^£fifellt>i_&o$j moving object is extracted from a dynamic 

^(ftl£ttt£ii"-5 image reliably and efficiently. 

[ftm^m [SOLUTION] 

^fyT'l 0 3T\ ^j-— if #\ At step 103, a user chooses two not continuous 

7=V 7^-f\"{ {c— K^^^ttfc re f erence frames out of the image of some 

lt©7U-i,©ItO^^ frames showed b V a list b y tne display. 

^ 'S^Lft^ 2*&-<£>K2i:7 u At Step 104 ' the contour J'ne of an objective 

\ ^T^+nJ-^ , ? = image is respectively extracted from the image 
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of each reference frame. 
After that, only in a suitable number, a user 
indicates the distinctive point on the 
contourjine of an objective image respectively 
in the image of each reference frame at step 
105, furthermore, if these are matched mutually, 
the contourjine of the image of the object in the 
intermediate frame between two reference 
frames will be presumed from the coordinates of 
the distinctive point on the image of each 
reference frame at step 106. 
And let this be a retrieval start contourjine. 
And it sets in the image of each intermediate 
frame, respectively, retrieval is started from a 
retrieval start contourjine, the contourjine of 
the image of an actual object is extracted. 
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101: Read image sequence 

102: List dynamic images 

103: Select reference frame 

104: Extract the outline in the reference frame 

105: Determine the corresponding point between the reference frames 

106: Extract the outline of the middle frame 

1 07: display the outline extract result 



[CLAIMS] 



[ft *if 1 ] 



[CLAIM 1] 

It is the image extracting method which extracts 
the contourjine of an objective image from the 
image of some frames in order, comprised such 
that image extracting method characterized by 
including the 1st step which receives 
designation of some distinctive points on the 
contourjine of the image of said object 
mutually corresponded between the images of 
two reference frames which are not followed of 
some frames, the 2nd step which computes the 
retrieval start line which starts retrieval of the 
contourjine of the image of said object in the 
image of the intermediate frame pinched by the 
reference frame of said 2 sheet based on the 
position of some distinctive points on the 
contourjine of the image of said object 
corresponded mutually between the images of 
each of said reference frame, respectively, and 
The 3rd step which starts retrieval of the 
contourjine of the image of said object from 
said retrieval start line in the image of the 
intermediate frame pinched by the reference 
frame of said 2 sheet, and extracts the 
contourjine of the image of said object in order 
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[CLAIM 2] 

It is the image extracting method which extracts 
the contourjine of an objective image from the 
image of some frames in order, comprised such 
that image extracting method characterized by 
including the 1st step which receives 
designation of some distinctive points on the 
contourjine of the image of said object 
mutually corresponded between the images of 
some reference frames which are not followed 
of some frames, the 2nd step which computes 
the retrieval start line which starts retrieval of 
the contourjine of the image of said object in 
the image of the intermediate frame pinched by 
the reference frame of order based on the 
position of some distinctive points on the 
contourjine of the image of said object 
corresponded mutually between the images of 
the reference frame of order, respectively, and 
The 3rd step which starts retrieval of the 
contourjine of the image of said object from 
said retrieval start line in the image of the 
intermediate frame pinched by the reference 
frame of order, and extracts the contourjine of 
the image of said object in order 



[CLAIM 3] 

It is the image extracting method of Claim 2, 
comprised such that said 2nd step is Image 
extracting method characterized by including 
the step which computes the interior division 
point which divides Between Some distinctive 
points on the contourjine of the image of said 
object contained in the image of one reference 
frame, and 

some distinctive points on the contourjine of 
the image of said object contained in the next 
reference frame of said one reference frame 
corresponded in each distinctive point of these 
distinctive points, into (N+1) which is the 
number 1 more than N which is respectively the 
number of the intermediate frames pinched 
between said one reference frames and next 
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reference frames of said one reference frame, 
and 

the step which makes the line which 
interpolated and obtained between the interior 
division points which are dividing between some 
distinctive points on the contourjine of the 
image of said object contained in the image of 
said one reference frame of said interior division 
points. 

and some distinctive points on the 
contourjine of the image of said object 
contained in the next reference-standard frame 
of said one reference-standard frame 
corresponded in each distinctive point of these 
distinctive points 

to J: (N-J +1) (J is N or less natural number) 
using the given interpolation function, as the 
retrieval start line of the intermediate frame Jth 
after said one reference frame. 



tit **f 3 ffitt©H«tttb^feT* 
Iff 3 IS«©M«tttb*ffiT* 



[CLAIM 4] 

It is the image extracting method of Claim 3, 
comprised such that said given interpolation 
function is a multinominal function low-orderer 
the first order than the number of said standard 
frames. 

Image extracting method characterized by the 
above-mentioned. 

[CLAIM 5] 

It is the image extracting method of Claim 3, 

comprised such that said given interpolation 

function is a spline function. 

Image extracting method characterized by the 

above-mentioned. 



[CLAIM 6] 

It is the image extracting method of Claim 1, 2, 

3, 4 or 5, comprised such that these frames are 

constituted from arbitrary positions and arbitrary 

directions by the photographed frame in said 

object using some cameras. 

Image extracting method characterized by the 

above-mentioned. 
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[DETAILED DESCRIPTION OF INVENTION] 



[00 0 1] 



[0001] 



[ISMroJR-f [TECHNICAL FIELD] 

Jf-WRte. WMU.<D^t)^h, W) Tnis invention relates to the image extracting 

^ (OW^^Wi^W(DM^.(D^m metnod which can extract the contourjine of 

85iBI&ffllllte:tttti1~5 ~ t&X the image of a ra P' dlv movin g object reliably out 

S SBttlttttl^jfeKiHI 1 *. ° f 3 dynamiC image - 



[0 0 0 2] 



[0002] 



it < f ij m £ tit v ^ SWMtff 
x - ^ a y&ffi t m-ffi x & 

V) ^ ^(D-^W:t LT, Hit <D 

ga^-r s n t «t o xnmttM 
ft&m*toibtix^z> 0 z.com 

^UH^mLXit, Computer 
Vision, Prentice-Hall, Inc. 1 9 
8 2 r % 4 $ Boundary 
Detection j ( D.H.Ballard, 
C.B.Brown M) i-Ett^tLTI^ 

yn-A#Jhlj^f (x, y) 



[PRIOR ART] 

Let it be a fundamental technique to extract the 
area of an analysis target object from a dynamic 
image in the dynamic image analysis utilized 
widely in various fields, such as a medical 
treatment and meteorology. 
Generally such a technique is called the image 
segmentation technique, as the way method, 
the changing point of the characteristics (for 
example, brightness etc.) of an image is 
detected as edge. 
It emphasizes. 
It connects. 

The contourjine of the image of an analysis 

target object is thus extracted. 

The limit detection is known. 

This limit detection is indicated by 

ComputerVision, Prentice-Hall, 

lnc.1982"Chapter 4 BoundaryDetection" (D. 

H.Ballard, C.B.Brown work). 

Hereinafter, about an edge detection 

characteristic of this kind of approach, the case 

where the monochrome still picture f (x y) is 

made into an object image is mentioned as an 

example, the edge detection indicated by this is 

demonstrated. 
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[0003] 

Fundamentally, the large place of the color 
between the adjacent pixels in an image or the 
difference of an image density is calculated as 
edge. 

An outline is caculated by connecting this. 
Here, the word "edge" is a line segment in a 
certain meaning which exists in an image. 
An outline becomes a closed curve, when the 
object showing the contour of the object which 
exists in an image does not hide into an image 
but is showing. 

Many edge exists in an outline top and other 
than that. 

Moreover, in the technique of a usual image 
processing conversely like the edge detection 
operator who states later, not all on an outline 
are recognized as edge. 



[0 0 04] 
[0 0 0 5] 

&mmt&mi%<Dm&, i, j m 

Si£r a ( i , j ) vmir a i ]f 



[0004] 

^-y v^^&Sfc&^i/^ti Being used since edge is calculated is as 
/tmj-v*.^ \. - j-. a — — - follows. 

[0005] 

In the case of monochrome concentration- 
difference image, i and j pixels are expressed 
with a (i, j). 

The direction of i corresponds in a horizontal 
direction. 

The direction of j corresponds in a vertical 
direction. 

[0006] 

(1) 

First derivation 

Horizontal direction: a(i+1, j)-a (i, j), vertical 
direction: a(i, j+1)-a (i, j), absolute value : 
The square root of the sum of Square of 
(a(i+1,j)-a (i,j) 
(Square of a(i, j+1)-a (i, j) 
etc. 



[0 0 0 6] 

(1) 

: a(i+1,j)-a(i,j), 
mtifa : a(i,j+1)-a(i,j), 
*6*Nif[ : (a(i+1,j)-a(i,j))©^i 
(a(i,j+1 )-a(i,j) <D ¥-15 <D f p <r> 

(2) -wm 

W5$\ : a(i+1,j)-2a(i,j)+a(i-1,j), 



(2) 
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S^fol : a(i,j+1)-2a(i l j)+a(ij-1), Second derivation 

Wi fp] (D ft : a(i+1j)+a(i- Horizontal direction: a(i+1, j)-2a(i, j)+a (i-1, j), 
1 j) + a(i,j+1)+a(ij-1)-4a(i,j) vertical direction: a(i, j+1)-2a(i, j)+a (i, j-1), the 

ft t\ Z T*S^©j^*-|B]Oft sum of bi-directionalities : 

a(i+1 ,j)+a(i-1 J)+a(i,j+1 )+a(i,j-1 )-4a(i,j) 

V72 ,_u ^Rte,^,^ p.-T-,^- . The sum of the last bi-directionahties may be 

v^ra, j^^fa, especially called Laplacian here. 

3fr3?iJCD;^<^-^£ffl^c This Laplacian TRIANGLE2 is computed in fact 

n ^aKI/ — >^^{dJ;o by the convolution calculation using the 

T^£b£ti<5 0 operator of the three-line three rows shown 

below. 

[0 0 0 7] [0007] 



0 1 o 

1 -4 1 
0 1 0 



yyy^ryv 2 it, That is, Laplacian TRIANGLE2 is expressed 
&)b LT\ i^co J; 5 Ci^^ti like following Formula as consequent. 



[0 0 0 8] 

V 2 =a (x, y - 1 ) + a (x 
- 1 , y) -4 a (x, y) 
+ a (x+1, y) +a (x, 
y+1) ... (IfciEU) 

ft 3 £ IJ CO h <D /g ^ Pj tl 6 fR "C 
tt&<, n^Tm^iJ (n, m(4 N 



[0008] 

TRIANGLE2=a(xy-1)+a(x-1, y)-4a (x y) 
+ a(x+1, y)+a (x y+1)... (Numerical formula 1) 
In addition, as an operator who uses for the 
convolution calculation performed when it is an 
edge detection, 3-row three column is not 
always used. 

The thing of n row m column (n and m are a 
natural number) may be used. 



[0 0 0 9] 
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J: By the way, the edge detected by such 
approach is interrupted under the influence of a 
noise etc. in many cases. 
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In order to connect the edge which interrupted 
to so, said Computer Vision, Prentice-Hall, and 
lnc.1982"Chapter 4 Boundary Detection" (D. 
H.Ballard, C.B.Brown work) smoothly, using the 
edge tracking method is indicated (p. 131 -p. 137 
reference). 



[0010] 

As an edge detection of those other than this, 

the image segmentation method described in 

Unexamined-Japanese-Patent No. 7-121710 

gazette is learned, for example. 

This image segmentation method is different 

from 

an edge detection described in said 
ComputerVision, Prentice-Hall, and 
lnc.1982"Chapter 4 BoundaryDetection" (D. 
H.Ballard, C.B.Brown work) 

The characteristics are to receive designation of 

the retrieval start pixel on the outline of the 

image of an analysis target object, and 

designation of a direction that edge retrieval is 

advanced, from a user side. 

It adds to this, according to an indication of a 

user, the contourjine which the user drew with 

the mouse etc. can be extracted as edge. 

This is also characteristic. 

That is, a user can give the pointer of edge 

retrieval intentionally. 

When connection of edge goes wrong, a user 
can determine the edge of an analysis target 
object intentionally. 

Therefore, only the edge of an analysis target 
object can be automatically extracted from the 
image in which edge became intricate intricately 
reliably markedly more efficiently than the case 
where edge retrieval is advanced arbitrarily. 
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[0011] 
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r SNAKES:Active Contour 
Models j 

( M.Kess.A.Witkin.D.Terzopoul 
os) International Journal of 
Computer Vision.vol.1 
No.4,1988 fdfeft ^ 

tc ft fi±fE«Mh Hit *> 3 i 
5/ ffl '£ PJ« ft^ft iC ft o X 



[0011] 

When, detecting the edge of an analysis target 
object from a dynamic image by the way, an 
approach is not used like the edge detection in 
said still picture about all the frames that 
constitute a dynamic image. 
In order to process a vast quantity of data 
efficiently, usually, it is indicated by 
"SNAKES:ActiveContourModels" (M. Kess, 
A.Witkin, D.Terzopoulos) 
InternationalJournalofComputerVision, vol 1 
No. 4, 1988. 

Only from the frame of the beginning of the 
frames which constitute a dynamic image, the 
edge of an analysis target object is detected 
according to the similar approach as the edge 
detection in said still picture. 
However, it is related with the frame which 
continues after that, it retrieves for edge by 
making into a retrieval starting point the edge of 
the analysis target object extracted from the last 
frame. 

The edge of an analysis target object is 
detected. 

The approach is adopted. 



[0 0 12] 



[0012] 



M] 
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[PROBLEM ADDRESSED] 

However, the approach adopted when detecting 
the edge of an analysis target object from said 
dynamic image had the fault that a detection of 
the edge of the intense analysis target object of 
a motion could not be coped with. 
There was the fault. 

That is, when a motion of an analysis target 
object is rapid, in a continuous frame, the 
position of an analysis target object is 
remarkably different mutually in many cases. 
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Therefore, it is also considering the edge of the 
analysis target object extracted from the last 
frame as a retrieval starting point, the edge of 
an analysis target object can not necessarily be 
detected. 



[0 0 13] 

o 



[0013] 

Then, this invention aims at providing the 
dynamic image extracting method which can 
extract the image of a rapidly moving object 
reliably and efficiently out of a dynamic image. 



[0 0 14] 



[0014] 



[SOLUTION OF THE INVENTION] 

In order to solve said subject, this invention is 
image extracting method which extracts the 
contourjine of an objective image from the 
image of some frames in order. 
Comprising: The image extracting method 
characterized by including 
The 1st step which receives designation of 
some distinctive points on the contourjine of 
the image of said object mutually corresponded 
between the images of two reference frames 
which are not followed of these frames, the 2nd 
step which computes the retrieval start line 
which starts retrieval of the contourjine of the 
image of said object in the image of the 
intermediate frame pinched by the reference 
frame of said 2 sheet, respectively based on the 
position of some distinctive points on the 
contourjine of the image of said object 
mutually corresponded between the images of 
each of said reference frame, and 
the 3rd step which starts retrieval of the 
contourjine of the image of said object from 
said retrieval start line in the image of the 
intermediate frame pinched by the reference 
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frame of said 2 sheet, and extracts the 
contourjine of the image of said object in order 
is provided. 



[0015] 



immtD^mmm] [Embodiment] 

WT\ 8§ft©EIE£:#$*Lft# Hereafter, one Embodiment based on this 

^^^i^6MM(D~MM invention >s demonstrated, referring attached 

iZ^xmW-tZo drawing. 
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[0016] 

First, FIG. 1 demonstrates the basic 
composition of the hardware of the image- 
processing system based on this embodiment. 



[0017] 

This image-processing system, the input device 
which receives input from a user, the image 
input device 308 which inputs digital dynamic 
image data for every frame, the display units 
305 which display a dynamic image (for 
example, a CRT monitor, a liquid crystal display, 
etc.), auxiliary memory 307 which stores the 
digital dynamic image data input from the image 
input device 308, the various program which 
defined the arithmetic processing performed by 
below-mentioned segmentation being under 
process, the processor 301 which performs the 
below-mentioned segmentation process, the 
buffer memory 302 for processor 301 , the frame 
memory 303 for processor 301 
And input-output COP 306 which connects 
auxiliary memory 307, an image input device 
308, and an input device 310,309 to a 
processor 301, the display controller 304 which 
converts into a RGB signal the image data 
stored in the frame buffer 303, and is output to a 
display unit 305, the system_bus 311 which 
connects these each part mutually 
are provided. 
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[0018] 

The display unit 305 which displays a dynamic 
image, has the characteristic (namely, each red, 
blue and green a 1000*1 000-pixel display, 256 
gradation, a 16,700,000 color display) made 
general as this kind of a display unit. 
When using the other display unit which has a 
different characteristic from this instead of this, 
, it is desirable to alter into the other image input 
device which has the capability which can 
correspond as required the image input output 
device 308 used now to the characteristic of a 
new display unit. 
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[0019] 

The input device which receives input from a 
user consists of mouse 310 grades which 
receive designation of the arbitrary positions on 
the keyboard 309 which receives a choice of the 
function assigned to each key, and the screen 
screen of a display unit 305. 
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[0020] 

Therefore, the user moved the cursor 
currently displayed on the screen screen of a 



05©7^!i ->mm±.\^7F dis P lav unit 305 usin 9 tne m ouse 310, 
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Furthermore, the predetermined button of the 
button (this embodiment two button) mounted 
on the mouse 310 is clicked. 
From the inside of some frames (refer FIG. 5) 
which constitute the dynamic image showed by 
a list in the state where it stood in a line on the 
screen screen of a display unit 305 at time 
series, the frame which constitutes a desired 
scene can be designated (after-mentioned). 
Hereafter, the mouse operation by the user at 
this time is called "the pick by the mouse 310." 
Moreover, the state where the button with which 
a "mouse-on state", and a call and this were 
conversely mounted on the mouse 310 is not 
pressed in the state where the button mounted 
on the mouse 310 is pressed will be called a 
"mouse-off state." 



[0021] 

In addition, in this embodiment, the mouse 310 
is used as an input device for designating the 
position on the screen screen of a display unit. 
However, it is not necessary to necessarily use 
a mouse, as substitution of a mouse 310, it may 
use other locator apparatus (for example, 
joystick etc.). 

Moreover, also about the operation method of a 
mouse 310, it is not necessary to adopt the 
operation method as demonstrated here, it is 
desirable to adopt the operation method which 
considered a user's operativity and added 
suitable alteration. 
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Moreover, a user chooses a suitable key out of 
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the key of a keyboard 309. 
The numerical value assigned to each key, the 
command which indicates execution of a 
program can be input. 



[0023] 

In the buffer memory 302 for processor 301, as 
shown in FIG. 2, the region 401 which stores 
predetermined data with starting of a system 
(namely, program region which stores the 
predetermined program stored in auxiliary 
memory 307), the operating-system region 
which stores the various program which 
constitutes an operating system (not shown), 
the data region 402 which stores the various 
data made necessary in the below-mentioned 
segmentation process 
are ensured. 



[0 0 2 4] 

^LX, /n;7^i«4 0 1 
l^*tix^Z>&m%H*. ^ 

m y* p if 7 a & t m-r 6 m « 4 
3 y^m^xmi^tizmm? 
mm? ^-As^itti/p ^ 7 

A^fB'lt1-5«4 0 4 % 

xmn $ ti 6 mmm&m * £ 

it L it ft $m m 7° d if =7 A £ 1 E 
ft-^S®«4 0 6 ^ ^5z&c^>-ir 



[0024] 

And each range included to the program range 
401 is assigned as 

The region 403 which stores the entire control 
program which respectively defined the flow of 
the below-mentioned segmentation process, 
the region 404 which stores the reference- 
standard frame outline extraction program 
which defined the reference-standard frame 
outline extraction process performed by below- 
mentioned segmentation being under process, 
the range 406 which stores the outline retrieval 
program which defined the outline retrieval 
process performed by below-mentioned 
segmentation being under process. 
And the range 405 which stores the retrieval 
start outline calculation program which defined 
the retrieval start outline calculation process 
performed by below-mentioned segmentation 
being under process 
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[0025] 

On the other hand, each range included to the 
data range 402 is assigned as 
The moving-image data range 407 which stores 
the digital moving-image data (or digital 
moving-image data input from the image input 
device 308) respectively stored in auxiliary 
memory 307, the contourjine data range 408 
which stores the contourjine data extracted by 
the outline retrieval process performed by 
below-mentioned segmentation being under 
process, the kernel pattern range 409 which 
stores the differential operator used for the 
convolution calculation of an edge detection. 
And workspace 410 which stores temporary 
data created in the below-mentioned 
segmentation process 
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[0026] 

Above, the description about the composition of 
the hardware of the image-processing system 
based on this embodiment is finished. 
In addition, in this embodiment, the information 
processor (a workstation, personal computer) of 
a stand-alone is used as a process part which 
corresponds to a processor 301, the buffer 
memory 302, and the input-output processor 
306. 

However, it is not necessary to necessarily do in 
this way. 

For example, a network system constitutes this 
image-processing system, it may make parallel 
perform the arithmetic processing in the below- 
mentioned segmentation process etc. in many 
computer resources. 

Moreover, it is not necessary to necessarily use 
the information processor used by this 
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embodiment, and the information processor 
(namely, information processor of a single bus 
architecture with which a processor 301, the 
buffer memory 302, and the input-output 
processor 306 share one system_bus) of the 
architecture of a same. 

For example, it may use the information 
processor which connected between a 
processor 301 and the input-output processors 
306 by the respectively separate system_bus 
between a processor 301 and the buffer 
memories 302. 



[0027] 

The segmentation process which a processor 
301 performs by FIG. 3, FIG. 4 hereafter 
according to the entire control program which 
the range 403 of the buffer memory 302 stored 
is demonstrated. 



[0028] 

If a user inputs a predetermined command 
using an input device, the segmentation 
process shown in FIG. 3 will be started. 
That is, in step 101, if the moving-image data 
range 407 of the buffer memory 302 stores the 
digital moving-image data (or digital moving- 
image data input from the image input device 
308) stored in auxiliary memory 307, the display 
controller 304 performed the predetermined 
image processing to the digital moving-image 
data which the moving-image data range 407 of 
the buffer memory 302 stored at this time. 
This is stored in a frame buffer 303. 
And after converting into a RGB signal the 
digital moving-image data stored in the frame 
buffer 303, it outputs to a display unit 305. 
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[0029] 

Consequently, it sets to step 102, on the screen 
screen of a display unit 305, the image of the 
frame which constitutes a dynamic image is 
showed by a list in the state where it ranked 
with time series, as shown in FIG. 5. 
In addition, it is not necessary to necessarily 
display the image of all the frames that 
constitute a dynamic image, for example, as 
long as it is the case where the blunt body of a 
motion is being comparatively used as the main 
photographed objects, it may show the image in 
every several frames by a list. 



[0030] 

After that, in step 103, if a user designates 
head frame F1 of a desired scene, and the 
image of the end frame Fn with "the pick by the 
mouse 310", respectively out of the image of the 
frame showed by a list on the screen screen of 
a display unit 305, head frame F1 and the end 
frame Fn which were designated at this time are 
set up as a reference-standard frame. 
After that, in step 104, a processor 301 
performs a reference-standard frame outline 
extraction process according to the reference- 
standard frame outline extraction processing 
program which the range 404 of the buffer 
memory 302 stored, that is, the edge detection 
demonstrated in the column of a PRIOR ART is 
performed using the differential operator whom 
the kernel pattern range 409 of the buffer 
memory 302 stored, the contourjine of the 
image of Object B is extracted from two 
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reference-standard frames F1 and the image of 
Fn, respectively. 

In addition, a reference-standard frame outline 
extraction process does not necessarily need to 
be based on the edge detection demonstrated 
in the column of a PRIOR ART, for example, it 
may be the range division known as another 
way method of a segmentation technique. 
And two reference-standard frames F1 and the 
contourjine data showing the contourjine of 
the image of the object B extracted from the 
image of Fn are stored to the contourjine data 
range 408 of the buffer memory 302, 
respectively. 

After that, the display controller 304 synthesized 
two reference-standard frames F1, the 
contourjine data showing the contourjine of 
the image of the object B extracted from the 
image of Fn, and two reference-standard 
frames F1 and the image data showing the 
image of Fn, respectively. 
The similar process as step 102 is performed, 
and it outputs to a display unit 305. 
Consequently, on the screen screen of a display 
unit 305, reference-standard frame F1 which 
carried out the synthesis display of the 
contourjines B1 and Bn of the image of the 
newly extracted object B, and the image of Fn 
are displayed. 
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[0031] 

After that, in step 105, as shown in FIG. 6, a 
user respectively indicates the contourjine B1 
of the image of Object B, the distinctive point on 
Bn (A1, An), and (N1, Nn) in two 
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reference-standard frames F1 and the image of 
Fn with "the pick by the mouse 310" by a 
suitable number of objects (n-1), if each 
distinctive point (A1, N1), ***, and (An, Nn) are 
matched mutually, in step 106, a processor 301 
performs the recursive process shown in FIG. 4, 
the contourjine of the image of Object B is 
extracted sequentially from two reference- 
standard frames F1, the intermediate frame F3 
between Fn(s), ***, and the image of Fn-1. 
Namely, a processor 301 performs the following 
processes in order by making the image of each 
intermediate frame F2, ***, Fn-1 into a 
process-target image. 



[0032] 

First, in step 201, a processor 301 performs 
retrieval start outline calculation according to 
the retrieval start outline calculation program 
indicated to the range 405 of a buffer memory, 
the coordinates of each distinctive point K1 on 
the contourjine of the object B contained in the 
image of the k-th intermediate frame Fk which is 
a new process target, ***, and Kn are 
presumed, furthermore, the digital line-segment 
description of between each presumed 
distinctive point K1 , ***, and Kn(s) is carried out, 
this is made into the retrieval start contourjine 
702 which should be made the origin which 
retrieves for the contourjine of Object B in the 
image of an intermediate frame Fk. 
(Refer FIG. 7). 

Specifically, as shown in the following, first, 
according to Numerical formula 2, the 
coordinates (Xt, Yt) (however, 1<t<n) showing 
each distinctive-point K1,***,Kn position on the 
outline of the object B contained in the image of 
the k-th intermediate frame Fk are presumed. 
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[0033] 

(Xt, Yt) ={(n-k) (X't, Y't)+ (k-1) 
(X"t, Y"t)}/(n-1)... (Numerical formula 2) 
here (X't, Y't) (X"t, Y"t) (however, 1<t<n)is 
are the coordinates showing each distinctive- 
point (A1,N1),***,(An,Nn) position which 
corresponds mutually between two reference- 
standard frames F1 and the outline of the object 
B contained in the image of Fn. 



[0034] 

Furthermore, it sets in two reference-standard 
frame F1 and the image of Fn, an edge 
detection respectively extracts contourjine 
B1,Bn point AAs.NNs which exists between 
adjacent distinctive-point (As,As-1),(Ns,Ns-1), it 
sets in the image of the k-th intermediate frame 
Fk using the coordinates of these two points 
AAs and NNs, and Numerical formula 2, the 
coordinates of the point (it is hereafter called an 
intermediate point) between adjacent 
distinctive-point Ks.Ks-1 on the contourjine of 
the image of Object b are presumed. 
And such a process is repeated, in order to 
create the high retrieval start contourjine of 
reliability, the coordinates of the intermediate 
point of sufficient number of objects are 
presumed. 

In addition, in this embodiment, the adjacent 
distinctive-point Ks,Ks-1 spacing etc. was 
considered, and the 7-piece intermediate point 
was presumed every between adjacent 
distinctive-point Ks,Ks-1. 
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After that, according to a digital line-segment 
description algorithm with a known DDA 
(digitaldifferentialanalyzer) algorithm etc., 
between the distinctive points K1, ***, Kn(s), 
and the intermediate points which were 
presumed is drawn, respectively, let this be the 
retrieval start outline 702 in the image of the k- 
th intermediate frame Fk (refer FIG. 7). 
In addition, the DDA algorithm mentioned as an 
example here is simplest algorithm as an 
algorithm which draws between detachment 
disseminations digitally. 

The detail is indicated by ComputerGraphics- 
principlesandpractice- and Addison- 
Wesley1990J. D.Foley et al. three-persons work 
( P 73-p74). 
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[0036] 

After that, a processor 301 is set to step 202, an 
outline retrieval process is performed according 
to the outline retrieval program which the area 
406 of a buffer memory stored. 
That is, it sets in the image of the k-th 
intermediate frame Fk, the point which exists 
really the retrieval starting point set up for every 
interval suitable on the retrieval start outline 702 
computed at step 201 on the contourjine Bk of 
the image of the actual object B from the image 
of the k-th intermediate frame Fk by performing 
an edge detection as one origin is extracted. 
In addition, what is sufficient is just to adopt the 
approach in which an approach etc. is known, 
as an edge detection performed at this time like 
"SNAKES:ActiveContourModels" (M. Kass, 
A.Witkin, D, Terzopoulos) 

IntwernationalJournalofComputerVision 
explained in the column of a PRIOR ART, vol.1 
No. 4, and the edge retrieval method indicated 
by 1988, for example. 

And a respectively similar process is performed 
about all retrieval starting points, the point 
which exists really on the contourjine of the 
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image of the contourjine of Object B is 
extracted in order. 

And between the extracted points is drawn 
according to known drawing algorithms, such as 
a DDA (digitaldifferentialanalyzer) algorithm, let 
this be the contourjine Bk of the image of the 
contourjine of the object B contained in the 
image of the k-th intermediate frame Fk (to refer 
FIG. 7). 

And the contourjine data showing this 
contourjine Bk are stored to the contourjine 
data range 408 of the buffer memory 302. 
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[0037] 

Now, a processor 301 performs the recursive 
process shown in FIG. 2, if it respectively 
finishes extracting contourjine B3,***,Bn-1 of 
the image of the contourjine of Object B from 
all intermediate-frame F2,***,Fn-1, in step 107, 
the display controller 304 synthesized the 
contourjine data showing the contourjine of 
the image of the object B extracted from each 
frame F1,***,Fn image which the contourjine 
data range 408 of the buffer memory 302 
stores, and the image data showing each 
F1,***,Fn image, respectively. 
The similar process as step 102 is performed, 
and it outputs to a display unit 305. 
Consequently, on the screen screen of a display 
unit 305, each frame F1,Fn image to which 
contourjine B1,***,Bn of the image of Object B 
was synthesized is showed by a list in the state 
where it ranked with time series. 
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[0038] 

Thus, the contourjine of the image of the 
object contained in the image of an intermediate 
frame is presumed from the contourjine of the 
image of the object contained in the image of 
two reference-standard frames, this is made 
into the origin of the retrieval in the image of 
each intermediate frame. 
Therefore, as shown in FIG. 7, in the image of 
each intermediate frame, retrieval can be 
started from the very neighbouring position of 
the contourjine of the image of a respectively 
actual object. 

That is, even if it is a rapidly moving object, the 
contourjine of the image can be extracted 
efficiently and reliably. 



[0039] 

Above, the description about the segmentation 
process based on this embodiment is finished. 



[0040] 

In addition, in this embodiment, two frames are 
set up as a reference-standard frame. 
However, it is not necessary to necessarily do in 
this way. 

For example, when an objective movement 
changes, a reference-standard frame is further 
set up into an intermediate frame, it may 
perform said segmentation process for each 
existing scene regular to objective movement. 
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Moreover, in this embodiment, when calculating 
the objective contourjine and objective search 
start outline of an image, drawing by the DDA 
algorithm is utilized. 

However, it is not necessary to necessarily 
utilize this, for example, it may utilize the digital 
linear drawing algorithm which draws a spline 
curve digitally. 

When three or more standard frames are set 
up, the order produces a suitable multinominal 
function low the first order rather than the 
number of sheets of a standard frame, it may 
make it utilize this as an interpolation function. 



[0042] 

In addition, as for dynamic image, it may 
image the same object simultaneously from a 
different direction from a different position using 
some cameras, this is synthesized. 
What was thus produced may be used. 
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[EFFECT OF THE INVENTION] 

According to the moving-image extracting 
method based on this invention 
the contourjine of the image of a rapidly 
moving object can be extracted from a dynamic 
image efficiently and reliably. 



[BRIEF EXPLANATION OF DRAWINGS] 



[Hi] [FIG.1] 

&%9l<DMM<DMWifci&%mi& 11 is the f '9 ure which showed the basic 

fcl^rAO^-K^xTro composition of the hardware of the image- 

l^fUjsfcSrTF LtzMX*fo& processing system based on the embodiment of 

0 this invention. 
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[FIG.2] 

It is the figure which showed the logical 
structure of the data space of the buffer memory 
of FIG. 1. 



[FIG.3] 

It is the flowchart which showed the flow of the 
segmentation process based on the 
embodiment of this invention. 

[FIG.4] 

It is the flowchart which showed the flow of the 
contourjine extraction process in an 
intermediate frame. 

[FIG.5] 

It is the figure which showed the image of the 
frame which comprises a dynamic image. 

[FIG.6] 

It is the figure which showed the distinctive point 
which corresponds between the images of a 
standard frame. 

[FIG.7] 

It is a figure for demonstrating the contourjine 
search process in an intermediate frame. 
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[EXPLANATION OF DRAWING] 

301... a processor 
302... a buffer memory 
303... a frame memory 
304... a display controller 
305... a display unit 
306... the input-output COP 
307... auxiliary memory 
308... an image input device 
309... a keyboard 
310... a mouse 
311... a bus 
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[FIG.1] 
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301: processor 
302: memory 
303: frame memory 
304: display controller 
305: display 

306: input-output control processor 

307: hard disk 

308: image input device 

309: key board 

310: mouse 

311: system bus 
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[FIG.2] 
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401: program memory 

403: the whole control program stored region 

404: reference frame outline extraction program stored region 

405: middle frame initial outline interpolation program stored region 

406: middle frame outline retrieval program stored region 

402: data memory 

407: dynamic image data region 

408: outline data region 

409: Kernel pattern region 

410: workplace 
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Frame number repeat 
201: determine retrieval outline 
202: retrieve outline points 
repeat on the outline point 
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[FIG.6] 
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reference frame F1, Reference frame Fn 
lm3] [FIG.3] 
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101: Read image sequence 

102: List dynamic images 

103: Select reference frame 

104: Extract the outline in the reference frame 

105: Determine the corresponding point between the reference frames 

106: Extract the outline of the middle frame 

107: display the outline extract result 
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[117] [FIG.7] 
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702: Retrieval starting outline 
Middle frame Fk 
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Derwent shall not in any circumstances be liable or responsible for the 
completeness or accuracy of any Derwent translation and will not be liable for any 
direct, indirect, consequential or economic loss or loss of profit resulting directly or 
indirectly from the use of any translation by any customer. 

Derwent Information Ltd. is part of The Thomson Corporation 

Please visit our home page: "WWW.DERWENT.CO UK" (English) 
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